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SPECIFICATION 
Sensor Fusion System Architecture 

Background of Invention 

[0001] This Application claims the benefit of U.S. Provisional Application No. 60/366,439, 
filed on 03/21/2002. 

[0002] 1 . Field of the Invention 

[0003] The present invention relates to a collision avoidance system of a vehicle, and more 
particularly to a collision avoidance system of a vehicle that is capable of distinguishing the type 
of objects around a vehicle and the position and velocity of the objects relative to the vehicle. 

[0004] 2. Description of the Prior Art 

[0005] There is a continuing increase in the density of vehicles on the road and a 
simultaneous emphasis on improving the safety of highway and ordinary roadway vehicles 
operation by preventing vehicles from colliding with stationary and moving objects. One means 
for accomplishing this is to detect or monitor the relative speed, the direction of travel, and the 
distance between vehicles sharing the roadway, and to use such information to provide direct 
indications to the vehicle's driver of potential danger. 

[0006] For example, US Patent 6,037,860 issued March 14, 2000 to Zander et al. and 
assigned to Volkswagen AG, teaches using a seat occupancy sensor to evaluate various collision 
strategies as an indicating function of possible severity of damage or injury. For this purpose, a 
collision strategy in which there are no injuries to persons is preferably selected, i.e. a collision 
with an inanimate object is carried out if this does not involve the driver and further vehicle 
occupants being subjected to excessive danger. For this purpose, the motor vehicle is actuated in 
such a way that most of the unoccupied areas of the vehicle absorb the impact energy, i.e. the 
vehicle is turned with the unoccupied areas towards the object with which it is colliding. 
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[0007] The Zander et al. system provides for a representation of the vehicle surroundings as 
determined by a number of laser sensors, radar sensors and thermal imaging and other cameras 
or a combination thereof. Thus, the Zander et al. system detects moving objects in the 
environment and internal vehicle conditions, in order to generate vehicle controls to avoid or 
minimize vehicle collisions. Sensor information is processed in an evaluation unit to determine a 
course of action and vehicle actuators implement the selected course of action. It would be 
desirable to have a more complete list of information regarding type of objects, distance from the 
host (including radial and angular information), and relative velocity. In particular, it would be 
desirable to have object-tracking data for visual awareness purposes, threat analysis, and possible 
subsequent vehicle actions. These are some of the problems the present invention overcomes. 

[0008] Further, the Zander et al. patent generally alludes to combining of various sensor 
technologies to form a picture of the targets surrounding the vehicle. However, for any particular 
target at any particular time, there is no teaching of using more than one sensor to obtain target- 
tracking information. It would be desirable to teach combining sensor information in an explicit 
way such that each field around the host car is covered in duplicate by radar and a camera. Using 
a fusion combination, in accordance with an embodiment of this invention, the best and most 
useful data from the radar is combined with the best and most useful data from the camera. 

[0009] US Patent No. 5,479,173 issued December 26, 1995 to Yoshioka et al. and assigned 
to Mazda Motor Corporation teaches combining radar and camera data to determine a danger 
level of an object in the predicted path of the host car. It would still be desirable to build a 
complete picture of objects around the host car. 

[0010] US Patent No. 5,475,494 issued December 12, 1995 to Nishida et al. and assigned to 
Mitsubishi Denki Kabushiki Kaisha teaches gauging size and distance of objects based on a 
calculated two-dimensional image. The system looks forward only and does not use true radar 
sensing. Data from a single forward looking camera are combined with data from a set of laser 
radar sensors. 
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Summary of Invention 

[001 1] This invention uses complementary sensors to provide information characterizing the 
environment around a vehicle. This information can be further processed and used in a variety of 
ways. For example, there may be implemented new advanced electronic features that use remote 
sensing in an integrated and cost-effective manner. Such features may include lane changing 
aids, night vision assistance, lane departure conditions, and other such features. 

[0012] Present solutions to providing information characterizing the environment around a 
vehicle often rely on single sensing technologies such as visual cameras, for lane departure, 
infrared (IR) cameras for night vision, and radar for adaptive cruise control. Such single sensing 
technologies are limited in robustness due to environmental and technological limitations. Such 
systems typically fail to take into account information gained by other sensors. 

[0013] In accordance with an embodiment of this invention, an electrical architecture 
integrates, or fuses, multiple sensors, and multiple sensing technologies, to create a data bank of 
information on objects surrounding the vehicle. Fused data can identify positions and velocities 
of objects, and can classify them in rough categories according to object size, such as heavy 
truck, passenger car, motorcycle, pedestrian, and so on. These data can then be communicated to 
all relevant control modules in the vehicle, to allow for more intelligent implementation of 
advanced features. It is particularly advantageous to combine sensors from two different bands of 
the electromagnetic spectrum. For example, visual cameras and radar sensors can be combined to 
achieve the above objective. Cameras and radars are located around the vehicle and sense 
overlapping areas. This allows distance data from radar to be combined with visual data from 
cameras. 

[0014] Thus, this invention teaches detecting the environment and providing a reliable set of 
data about objects in the environment. Radar and camera sensors can provide the inputs to a 
central fusion module. The central fusion module processes the inputs to generate object-tracking 
data. This includes a list of information regarding type of object, distance from host car, 
including radial and angular information, and velocity. The object tracking data are available as 
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output from the central fusion modules for visual awareness purposes, threat analysis, and 
possible vehicle actions. 

[0015] In accordance with this invention, the sensors are arrayed such that each field around 
the host car is covered in duplicate by radar and a camera. The combining of the data is then 
done by fusion. That is, the best and most useful data from the radar is combined with the best 
and most useful data from the camera. Using both sensing technologies provides an improvement 
over using a single sensing technology alone to determine necessary information about 
surrounding objects, whether moving or stationary. This type of data fusion requires a system 
that provides a time tag with the data so that the data can be combined effectively. If desired, the 
data can be interpolated and extrapolated to the point in time for the fusion to provide increased 
accuracy. These fused data are stored in a target track file. Subsequent additional fused data are 
compared to the current track files and if a match is found the track file is updated. The current 
track files are output for further processing. 
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Brief Description of Drawings 

[0016] Figure 1 is a plan view of a vehicle with information gathering zones A, B, C, and D, 
each zone including two sensors. 

[0017] Figure 2 is a block diagram showing a collision avoidance system of a vehicle 
according to one embodiment of the present invention. 

[0018] Figure 3 is a block diagram showing a more detailed view of the input sensors shown 
in Figure 2. 

[0019] Figure 4 is a block diagram of the function of the fusion control module as 
information from the sensors is processed to obtain target definition for multiple targets using 
matching of time tagged data. 

[0020] Figure 5 is a block diagram of the function of the fusion control module as 
information from the sensors is processed to obtain target definition for multiple targets using 
extrapolation and interpolation of time tagged data. 
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Detailed Description 

[0021] Figure 1 is a plan view of a vehicle 1 with illustrated sensing zones A, B, C, and D. In 
each of zones A, B, C, and D two sensors are deployed. The sensors are each from different 
bands of the electromagnetic radiation spectrum and provide complementary information. 
Advantageously, one is selected to be an active radar sensor and the other a visual sensor, such 
as a camera. Thus, there are two overlapping sensors in each of the zones A, B, C, and D. 
Because the sensors overlap complementary information is available about any target in the 
zone. The two sensors can provide more information about the target than either sensor could 
individually. More specifically, zone A includes a camera sensor 110 and a radar sensor 210, 
zone B includes a camera sensor 111 and a radar sensor 211, zone C includes a camera sensor 
1 12 and a radar sensor 212, and zone D includes a camera sensor 113 and a radar sensor 213. 

[0022] Information from these camera sensors 110, 111, 112 113 and radar sensors 210, 21 1, 
212, 213 is processed as complementary pairs to determine information about the environment 
surrounding the vehicle. Information from the sensors, camera sensors 110, 111, 112 113 and 
radar sensors 210, 211, 212, 213, is time tagged to indicate the time that the information was 
gathered. In accordance with an embodiment of this invention, information from paired camera 
sensor 110 and radar sensor 210 in zone A with the closest time tags is compared and fused to 
provide fused information data. This information is then used to generate characterizing 
descriptions of any target in the particular zone generating the sensor signals. Similarly, 
information from sensor pairs in zone B (camera sensor 111 and radar sensor 211), zone C 
(camera sensor 1 12 and radar sensor 212), and zone D (camera sensor 113 and radar sensor 213) 
is time tagged and fused to provide fused information data. For example, radar sensor 210 and 
camera sensor 110 in zone A can provide information about a motorcycle in zone A. Such 
information can include size, speed, direction, acceleration, and so on. 

[0023] Radar sensors are particularly advantageous to help determine distance and whether a 
target is moving or standing still. Relative velocity can be determined based on the frequency 
shift of the received radar signal versus the transmitted radar signal. Cameras can provide good 
information with respect to angular information and object recognition. Thus the complementary 
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combination of a camera and radar sensor provides more information than either sensor could 
individually. 

[0024] Figure 2 is a block diagram showing a vehicle collision avoidance system 10 of a 
vehicle, according to a preferred embodiment of the present invention. Collision avoidance 
system 10 includes an input detecting section 11 having an output coupled to a central fusion- 
processing module 13. The output of central fusion module 13 is applied to a vehicle control 
module 15. Central fusion module 13 supplies an activating output to vehicle control module 15, 
which may include a plurality of vehicle control modules. In situations where a crash may be 
imminent, an active safety system may be energized and other pre-crash action may be taken. 
Thus an active and a passive safety module may be activated. In turn, these modules may control 
a braking module, a light control module, an adaptive steering module, a security module, and 
other modules. 

[0025] More specifically, referring to Figure 3, input-detecting section 11 includes four 
visual cameras 1 10, 1 1 1, 1 12, and 113 and four radar sensors 210, 211, 212, and 213. There is an 
output from the combination of sensors in input detecting section 1 1 to central fusion module 13. 
The visual cameras give good angular information about adjacent objects. The radar sensors give 
good depth and distance information about adjacent objects. 

[0026] Referring to Figure 4, input-detecting section 11 provides inputs to central fusion 
module 13. Central fusion module 13 includes a tag block 21 coupled to receive an output from 
input-detecting section 1 1 for adding a time tag to the incoming sensor signal. Thus, each sensor 
signal is identified with respect to the time the sensor signal is received. Tag block 21 has an 
output coupled to the input of a matching block 22. Matching block 22 matches the time tagged 
visual and camera sensor inputs to a time tagged radar sensor input. Advantageously, this is done 
so that the closest time tags of each of the visual and radar inputs are matched. The output of 
matching block 22 is applied to a target definitions block 23. 

[0027] Thus, referring to Figure 4, a time tag indicating the time the data is generated is 
added to the input data received from the input sensors. Advantageously, some information from 
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all the input sensors is used. A qualitative determination is made with respect to the value and 
accuracy of the information from each sensor. The best-input data is weighted more heavily than 
input data of lesser reliability. Thus, for each target a time tagged compilation of input data is 
used to characterize the target with respect to position, velocity, acceleration, size, and so on. 
This same procedure of using data from all sensors is applied to all targets in the surrounding 
environment. Further, for each target tracking data, the process is repeated as time goes on so 
that an updated target track is continually generated. 

[0028] Referring to Figure 5, input-detecting section 11 provide inputs to central fusion 
module 13a. Central fusion module 13a receives an input at a tag block 21a. Tag block 21a has 
an output coupled to an interpolation/extrapolation block 22a. Interpolation/extrapolation block 
22a processes information to get time/target information of increased accuracy at those points in 
time when actual time tagged sensor information indicates a slightly different time. To 
compensate for these slight differences in time the information from a sensor pair is interpolated 
and/or extrapolated to generate information characterizing the environment around the vehicle at 
substantially the same time. That is, after the data is time tagged it can be used to interpolate 
target information between time tags, and can be extrapolated to indicate projected target 
information after a time tag. The output of interpolation/extrapolation block 22a is coupled to 
target definitions block 23a. 

[0029] In summary, a fusion process combining complementary data is used to characterize 
each target. The fusion process uses data from different sensors at the same point in time to get 
better definition of target objects surrounding the vehicle. The information characterizing each 
target is then combined to generate a picture of the environment surrounding the vehicle. Such a 
picture would include all the targets, in all directions from the vehicle. Thus, the system provides 
a time tag with the data so it can be extrapolated to the same point in time for the fusion process 
to take place. The fused data can be stored in a target track file. Additional subsequent fused data 
can be compared to the current track files and if a match to a target is found the track file is 
updated. The current track files can be output for further processing. 
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[0030] Target-tracking information is obtained from all sensors of this embodiment. Each 
piece of information from a sensor relating to a particular target is given a time tag to identify it 
in the time domain. A plurality of such target tracking pieces of information is used through 
interpolation and extrapolation to obtain additional target tracking pieces of information. Such 
information from all sensors is fused together to get the best representation of the target position 
at a given time. This process is repeated for all targets around the 360-degree circumference of 
the vehicle. The totality of these targets is then combined to form a total picture of targets 
surrounding the vehicle. 

[0031] While the present invention has been particularly shown and described with reference 
to preferred embodiment thereof, it will be understood by those skilled in the art that the 
foregoing and other changes in form and details may be made therein without departing from the 
spirit and scope of the present invention. For example, infrared sensors and laser sensors may be 
selected for use from the electromagnetic spectrum. Also, the time tagging function may occur at 
the sensor if the central fusion module has distributed processing ability. 



